STSM: Spatio-Temporal Shift Module for Efficient Action Recognition

The modeling, computational complexity, and accuracy of spatio-temporal models are the three major foci in field video action recognition. traditional 2D convolution has low but it cannot capture temporal relationships. Although 3D can obtain good performance, is with both high complexity a large number parameters. In this paper, we propose plug-and-play Spatio-Temporal Shift Module (STSM), which effective high-performance module. STSM be easily inserted into other networks to increase or enhance ability network learn features, effectively improving performance without increasing parameters complexity. particular, when CNNs integrated, new may features outperform based on convolutions. We revisit shift operation from perspective matrix algebra, i.e., sparse kernel. Furthermore, extensively evaluate proposed module Kinetics-400 Something-Something V2 datasets. experimental results show effectiveness STSM, recognition also achieve state-of-the-art two benchmarks.